Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(java/python): new xlang type system spec implementation #1690

Merged
merged 79 commits into from
Dec 30, 2024

Conversation

chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Jun 16, 2024

What does this PR do?

This PR implements a new type system for xlang serialization between java and python.

The changes includes:

  • Refine type system spec: added new types:
    • named_enum: an enum whose value will be serialized as the registered name.
    • struct: a morphic(final) type serialized by Fury Struct serializer.
    • polymorphic_struct: a type which is not morphic(not final). i.e. it don't have subclasses. Suppose we're deserializing
      List<SomeClass>, we can save dynamic serializer dispatch if SomeClass is morphic(final).
    • compatible_struct: a morphic(final) type serialized by Fury compatible Struct serializer.
    • polymorphic_compatible_struct: a non-morphic(non-final) type serialized by Fury compatible Struct serializer.
    • named_struct: a struct whose type mapping will be encoded as a name.
    • named_polymorphic_struct: a polymorphic_struct whose type mapping will be encoded as a name.
    • named_compatible_struct: a compatible_struct whose type mapping will be encoded as a name.
    • named_polymorphic_compatible_struct: a polymorphic_compatible_struct whose type mapping will be encoded as a name.
    • ext: a type which will be serialized by a customized serializer.
    • polymorphic_ext: an ext type which is not morphic(not final).
    • named_ext: an ext type whose type mapping will be encoded as a name.
    • named_polymorphic_ext: an polymorphic_ext type whose type mapping will be encoded as a name.
  • Added a new XtypeResolver in java to resolve xlang types
  • Support register class mapping by id. Before this PR, we only support register class by name, which is more expensive at space/performance cost.
  • Support pass type into to resolve type ambiguation such as ArrayList/Object[] in java. Users can serialize(List.of(1, 2, ,3)) and deserialize it into array by deserialize(bytes, Integer[].class)
  • Refactor pyfury serialization by moving type resolver into python code from cython, this will make debug more easy and reduce code duplciation, it also speed serialization performance.
  • golang xtype serialization test are disabled, it will be reenabled after new type system is implemented in golang

Related issues

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

@chaokunyang chaokunyang marked this pull request as draft June 16, 2024 14:40
chaokunyang added a commit that referenced this pull request Jul 13, 2024
## What does this PR do?

This PR moves `org.apache.fury.reflect.Types` into TypeRef. Types is
used in Fury type system, by merge `org.apache.fury.reflect.Types`, we
can reduce ambiguation in fury type system

## Related issues
#1553 
#1690 

## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fury/issues/new/choose) describing the
need to do so and update the document if necessary.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?


## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
@chaokunyang chaokunyang force-pushed the new_xlang_sepc_impl branch from 86a35c3 to a3d4d9d Compare July 13, 2024 11:41
@chaokunyang
Copy link
Collaborator Author

Hi @theweipeng @pandalee99 @penguin-wwy @PragmaTwice @jiacai2050 , this PR is ready for review, could you help review it?

Copy link
Contributor

@pandalee99 pandalee99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR is basically okay.
It supports new types and improves the specifications of the type system. For instance, the XtypeResolver class has established a rather intricate yet fully functional mechanism for type resolution, registration, and the processing of related information.This PR has added a lot of functions and optimized the performance.

@pandalee99 pandalee99 merged commit 98efd72 into apache:main Dec 30, 2024
37 checks passed
pandalee99 pushed a commit that referenced this pull request Jan 31, 2025
…n in pyfury (#2034)

## What does this PR do?

This pr unifies type system between python and xlang serialization in
pyfury, so that we can remove duplicate code in pyfury, and lay the
foundation for following features:
- [ ] implement protocol such as chunk based map serialization for
pythobn and xlang serialization
- [ ] use python exsiting fastpath optimization in xlang serialization
- [ ] extend `DataClassSerializer` to support codegen based
serialization for xlang

## Related issues
#1690

## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fury/issues/new/choose) describing the
need to do so and update the document if necessary.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?

## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants